{ "cells": [ { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "Zk-GvtlVYJln" }, "source": [ "
\n", "\n", "# Denoising acquired images using deep learning\n", "\n", "Copy this notebook into a directory in your Google drive. \n", "\n", "In this tutorial we will create a deep learning denoising model trained on data aquired by Pycro-Manager on your microscope. We will then used this denoising model to denoise images collected by Pycro-Manager in real time. \n", "\n", "We will be running image aquisition and inference locally, and train on a Google Colab GPU instance, though if you have a fairly powerful GPU locally feel free to train locally.\n", "\n", "The deep learning model used in this tutorial is [N2V](https://github.com/juglab/n2v), which allows us to create a denoising algorithm without groud truth images by training on noisy images without clean targets. Check out how it works [here](https://ieeexplore.ieee.org/document/9098336) and [here](https://arxiv.org/pdf/1811.10980.pdf).\n", "\n", "Please install [Pycro-Manager](https://pycro-manager.readthedocs.io/en/latest/setup.html) locally before running this Colab notebook.\n", "\n", "Written by [Ryan Mei](https://github.com/rmeit), [Henry Pinkard](https://github.com/henrypinkard)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "EHs5GbfqZ4-s" }, "source": [ "
\n", "\n", "## Part 1: Connect to a local runtime\n", "\n", "Open this notebook on your local computer. If you have not yet, install Pycro-Manager: " ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "!pip install pycromanager" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Open micromanager and connect your microscope to your computer." ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "KSsbKOGiDZuI" }, "source": [ "
\n", "\n", "## Part 2: Collecting Training Images\n", "\n", "First verify you have a working installation of [Pycro-Manager](https://pycro-manager.readthedocs.io/en/latest/setup.html). Open Micro-Manager, select tools-options, and check the box that says Run server on port 4827 (you only need to do this once). Run:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "colab": {}, "colab_type": "code", "id": "Qoz8ovU4DgP3" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n" ] } ], "source": [ "import matplotlib.pyplot as plt\n", "import numpy as np\n", "\n", "from pycromanager import Core\n", "\n", "core = Core()\n", "print(core)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "8py8Ym6oD7h9" }, "source": [ "The output should look something like: \n", "```\n", "Out[1]: JavaObjectShadow for : mmcorej.CMMCore\n", "```" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "E8nCu7TKdD1W" }, "source": [ "It is important that the images we use to create the denoising use the same camera and imaging settings (gain, em-gain, read-out-parameters,...) as in your experiments. \n", "We recommend that you aquire 3-10 images. If your camera is higher resolution, or if you are running this notebook without a GPU, it sometimes take more than 12 hours to train. In this tutorial we will be capturing images of a single scene though you may gain improved performance from capturing different samples and fields of view." ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "VlS1SM2CYJmd" }, "source": [ "### Aquisition\n", "\n", "Adjust your microscope to the imaging settings (gain, read-out-parameters...) you plan to use in your experiments. Stage your sample. We will now collect the images and store them in a numpy array. \n", "Let's first try snapping a single image using the cell below. Make any adjustments needed." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 287 }, "colab_type": "code", "id": "4ZzmQXuKI4NW", "outputId": "fb7d77c1-49ad-43e4-d7b7-0ae2071d7474" }, "outputs": [], "source": [ "## Optional: Set microscope properties here. Here we set a property of\n", "## the core itself, but same code works for device properties\n", "# auto_shutter = core.get_property('Core', 'AutoShutter')\n", "# core.set_property('Core', 'AutoShutter', 0)\n", "\n", "core.snap_image()\n", "tagged_image = core.get_tagged_image()\n", "pixels = np.reshape(\n", " tagged_image.pix, newshape=[tagged_image.tags[\"Height\"], tagged_image.tags[\"Width\"]]\n", ")\n", "plt.imshow(pixels, cmap=\"magma\")" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "colab": {}, "colab_type": "code", "id": "zL9wd8a5hfMU" }, "outputs": [], "source": [ "quantity = 6 # Adjust to number of images you would like to collect\n", "dataRaw = []\n", "\n", "\n", "def snap_and_get_image():\n", " core.snap_image()\n", " tagged_image = core.get_tagged_image()\n", " pixels = np.reshape(\n", " tagged_image.pix,\n", " newshape=[tagged_image.tags[\"Height\"], tagged_image.tags[\"Width\"]],\n", " )\n", " dataRaw.append(pixels)\n", " quantity -= 1\n", "\n", "\n", "while quantity >= 0:\n", " snap_and_get_image()\n", "\n", "dataRaw = np.array(dataRaw)\n" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "sQ4I2_PXyUAs" }, "source": [ "Let's save our data in a numpy array." ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "colab": {}, "colab_type": "code", "id": "BLQuyEj9yXxw" }, "outputs": [], "source": [ "np.save(\"dataRaw.npy\", dataRaw)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "BghrcoaCpkJ5" }, "source": [ "
\n", "\n", "## Part 3: Creating the Model\n", "\n", "If you have a Nvidia GPU and would like to train the model locally, feel free to skip to the next block of code. Otherwise, we will connect to a Colab runtime to utilize a free GPU instance. \n", "\n", "We want to enable GPU acceleration to speed up training. Under 'Runtime' dropdown in the left top bar, select 'change runtime type' and select 'GPU'. \n", "\n", "First disconnect from the local runtime using the dropdown in the top right, and switch the runtime back to 'hosted'." ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 148 }, "colab_type": "code", "id": "RagzNkkls1Jm", "outputId": "fdddcfe7-c902-4281-e848-e692067f6f75" }, "outputs": [], "source": [ "% tensorflow_version 1.x # CSBDeep is built on Tensorflow v1 and will NOT work with v2.\n", "% nvidia-smi # Check that we are connected to a GPU\n", "import tensorflow as tf\n", "\n", "print(\"Num GPUs Available: \", len(tf.config.experimental.list_physical_devices(\"GPU\")))" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 369 }, "colab_type": "code", "id": "ieAgRzhIsQkW", "outputId": "7f304ffd-6fa5-4c6c-d0f5-a243262f6551" }, "outputs": [], "source": [ "% pip install n2v # Install noise2void\n", "import os\n", "\n", "import numpy as np\n", "from matplotlib import pyplot as plt # Libraries for plotting\n", "from matplotlib.image import imread, imsave # For processing images\n", "from n2v.internals.N2V_DataGenerator import N2V_DataGenerator\n", "from n2v.models import N2V, N2VConfig\n", "from PIL import Image" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "8tYBJ86MsPNX" }, "source": [ "Upload `dataRaw.npy` from the directory you started your local runtime to the Colab notebook directory using the 'file' button on the left side-menu." ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "colab": {}, "colab_type": "code", "id": "Hst_5ut9rVjS" }, "outputs": [], "source": [ "imgs = np.load(\"dataRaw.npy\")\n", "testImg = imgs[0]\n", "for im in range(1, len(imgs)):\n", " Image.fromarray(imgs[im]).save(str(im) + \".tif\")\n", "imgs = N2V_DataGenerator.load_imgs_from_directory(\"/content\", filter=\"*.tif\", dims=\"YX\")\n", "# If on local runtime use:\n", "# imgs = np.load('dataRaw.npy')" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "Q-Z_k9nnuPEe" }, "source": [ "Check that we can view our images." ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 425 }, "colab_type": "code", "id": "cE2ktf73uN2H", "outputId": "f4cf05f7-9900-45a0-b6ac-ca5a44ba7d2a" }, "outputs": [], "source": [ "plt.figure(figsize=(14, 7))\n", "plt.subplot(1, 2, 1)\n", "plt.imshow(imgs[0][0, ..., 0], cmap=\"magma\")\n", "plt.subplot(1, 2, 2)\n", "plt.imshow(imgs[6][0, ..., 0], cmap=\"magma\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We will now create training patches and validation patches from the images we collected. Feel free to change the shape as N2V can train off of abitrarily large patches." ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 202 }, "colab_type": "code", "id": "UnRxhuuJwoKX", "outputId": "012f6490-702f-4a39-ce7a-d4fa93f9ea29" }, "outputs": [], "source": [ "patches = []\n", "patches = N2V_DataGenerator.generate_patches_from_list(\n", " imgs, shape=(64, 64), shuffle=True\n", ")\n", "\n", "divide = int(len(patches) / 8)\n", "train_patches = patches[divide:]\n", "val_patches = patches[:divide]" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "lclT96evyOmV" }, "source": [ "Let's look at one of our training and validation patches." ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 419 }, "colab_type": "code", "id": "FXxbSX6yyLtb", "outputId": "c4733aeb-1867-4721-b89f-8f3330cb3cba" }, "outputs": [], "source": [ "plt.figure(figsize=(14, 7))\n", "plt.subplot(1, 2, 1)\n", "plt.imshow(train_patches[0, ..., 0], cmap=\"magma\")\n", "plt.title(\"Training Patch\")\n", "plt.subplot(1, 2, 2)\n", "plt.imshow(val_patches[0, ..., 0], cmap=\"magma\")\n", "plt.title(\"Validation Patch\");" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "8kEQ1ItHyvUc" }, "source": [ "Let's configure our model. We very strongly recommend that you not train for more than 120 epochs on Colab as the system will time out after 12 hours. Make sure not to close your browser or after 90 minutes this notebook's data will be erased. For detailed documentation of the parameters and what they mean, check [this](https://github.com/juglab/n2v/blob/master/n2v/models/n2v_config.py) out." ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 497 }, "colab_type": "code", "id": "tJJ5HwOnyv6F", "outputId": "dabb12b0-7cc8-459d-ba65-2683b3a96856" }, "outputs": [], "source": [ "config = N2VConfig(\n", " train_patches,\n", " unet_n_depth=3,\n", " unet_kern_size=3,\n", " train_steps_per_epoch=300,\n", " train_epochs=80,\n", " train_learning_rate=0.0005,\n", " train_loss=\"mse\",\n", " batch_norm=True,\n", " train_batch_size=128,\n", " n2v_perc_pix=0.198,\n", " n2v_patch_shape=(64, 64),\n", " unet_n_first=96,\n", " unet_residual=True,\n", " n2v_manipulator=\"uniform_withCP\",\n", ")\n", "vars(config)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "4y5gKKt51H3v" }, "source": [ "Mount Google Drive. We will save our model to a folder in Drive to not loose it when we close this Colab notebook." ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "colab": {}, "colab_type": "code", "id": "6bf4XM251I1C" }, "outputs": [], "source": [ "from google.colab import drive\n", "\n", "drive.mount(\"/content/gdrive\")" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 517 }, "colab_type": "code", "id": "BLCSOZLDzZME", "outputId": "bd691e9e-4a2f-4c18-94f6-bb88cd113d33" }, "outputs": [], "source": [ "model_name = \"n2v_fluorescence_microscopy\"\n", "model_dir = \"/content/gdrive/My Drive/denoising_model\"\n", "# We are now creating our network model.\n", "model = N2V(config=config, name=model_name, basedir=model_dir)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "oAtRm1Dczcro" }, "source": [ "Time to train our model. Make sure not to close this notebook during training if you are using a hosted runtime. This may take a while, around 11 hours for 100 epochs with the provided settings." ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 813 }, "colab_type": "code", "id": "ANmwZIIZzoZ9", "outputId": "20ad8cef-faca-43e6-ff72-6a70fc1feadb" }, "outputs": [], "source": [ "history = model.train(train_patches, val_patches)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "7XTENRnK0Yzm" }, "source": [ "Let's test our fresh model on an image we collected earlier." ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 437 }, "colab_type": "code", "id": "7hMWoT8g0ZBg", "outputId": "c575aa59-936b-45a2-e2a8-bed48c950c68" }, "outputs": [], "source": [ "pred = model.predict(testImg, axes=\"YX\")\n", "plt.figure(figsize=(14, 7))\n", "plt.subplot(1, 2, 1)\n", "plt.imshow(testImg, cmap=\"magma\")\n", "plt.title(\"Raw Image\")\n", "plt.subplot(1, 2, 2)\n", "plt.imshow(pred, cmap=\"magma\")\n", "plt.title(\"Denoised\");" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "f9gfmMYZ2ArK" }, "source": [ "
\n", "\n", "## Part 4: Testing our Model\n", "Now let's test using our algorithm and denoise images collected in real time using Pycro-Manager! \n", "\n", "First, start and reconnect to a local runtime. Download the folder `denoising_model` from your Google Drive to the current working directory of your local runtime. \n", "\n", "Let's load our model." ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "colab": {}, "colab_type": "code", "id": "RlYbrBJP3JBJ" }, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "import numpy as np\n", "from n2v.models import N2V, N2VConfig\n", "\n", "from pycromanager import Acquisition, multi_d_acquisition_events\n", "\n", "model_name = \"n2v_fluorescence_microscopy\"\n", "basedir = \"./\"\n", "# We are now creating our network model.\n", "model = N2V(config=None, name=model_name, basedir=basedir)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "1AESOfKnTgmC" }, "source": [ "Create a Pycro-Manager [image processor](https://pycro-manager.readthedocs.io/en/latest/img_processors.html) that applies the deep learning model we created to images." ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "colab": {}, "colab_type": "code", "id": "hGA-evviT7kI" }, "outputs": [], "source": [ "def img_process_fn(image, metadata):\n", " # Apply our algorithm to the collected image and return the result\n", " image = model.predict(img, axes=\"YX\").astype(np.uint16)\n", " return image, metadata" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "6dD1XKPPZ-3O" }, "source": [ "Let's aquire an image!" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "colab": {}, "colab_type": "code", "id": "P5avXUswTQAT" }, "outputs": [], "source": [ "directory_to_save_images = \"/aquisitions_tmp\"\n", "with Acquisition(\n", " directory=directory_to_save_images,\n", " name=\"acquisition_1\",\n", " image_process_fn=img_process_fn,\n", ") as acq:\n", " events = multi_d_acquisition_events(num_time_points=10)\n", " acq.acquire(events)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "UxbsxrfYaCeX" }, "source": [ "To learn how to read your denoised images [check this out](https://pycro-manager.readthedocs.io/en/latest/apis.html#reading-acquired-data)!" ] } ], "metadata": { "accelerator": "GPU", "colab": { "name": "pycro_manager_denoising_demo_n2v.ipynb", "provenance": [] }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.6" } }, "nbformat": 4, "nbformat_minor": 4 }